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ABSTRACT 



Governments and other stakeholders have become increasingly interested in assessing the s ki lls 
of their adult populations for the puiposes of monitoring how well prepared they are for the challenges of 
the new information world. The current paper provides an overview of the conceptual framework 
developed for the assessment of problem solving in technology-rich environments for the OECD’s 
Programme for the International Assessment of Adult Competencies (PLAAC). This covers the specific 
class of problems that people encounter when using information and communication technologies. These 
include problems where the existence of the problem is a consequence of the availability of new 
technologies, where the solution requires the use of computer-based applications or where the problem 
relates to the management or use of information technologies. 



RESUME 



Dans la region OCDE tout comme en dehors, les pouvoirs publics et autres parties prenantes 
s’interessent de plus en plus a T evaluation des competences de la population adulte dans un objectif de 
suivi de son etat de preparation face aux defis de la societe moderne de la connaissance. Le present 
document fournit une vue d’ensemble du cadre conceptuel en matiere de resolution de problemes dans un 
environnement hautement technologique du Programme pour P evaluation intemationale des competences 
des adultes (PLAAC). L’evolution et le role des technologies de l’information dans la societe d’aujourd’hui 
y sont presentes et la definition de la resolution de problemes et les constructions mentales requises dans ce 
contexte font l’objet d’une analyse. Le cadre presente egalement les dimensions cles et taches prevues en 
matiere de resolution de problemes, avant de souligner certains aspects pratiques de P evaluation. 
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PIAAC PROBLEM SOLVING IN TECHNOLOGY-RICH ENVIRONMENTS: A CONCEPTUAL 

FRAMEWORK 



INTRODUCTION 



Structure of the present document 

1. This document introduces the domain of problem solving in technology-rich environments in 
PIAAC and explains how the domain will be assessed. The document contains four parts. Part 1 discusses 
the development of information technologies and their uses in today’s societies. Part 1 also introduces 
various constructs of problem solving, especially as they apply in technology-rich environments. Part 2 
provides a definition of problem solving in technology-rich environments and elaborates on some of the 
assumptions underlying this definition. Part 3 presents the core dimensions that will be represented in the 
tasks to be included in the assessment. Part 3 also contrasts the new problem solving in technology-rich 
environments (PS-TRE) domain with the other domains assessed in PIAAC. as well as with other 
constructs of technology knowledge and use. Finally, Part 4 reviews some practical aspects of the 
assessment, especially regarding the evaluation of problem solving strategies in the context of digital 
technology. 

Impact of digital technologies on instruction and assessment 

2. During the past decades, digital technologies have deeply transformed the way individuals learn, 
communicate, work, and more generally the way they function in societies. Shopping, traveling, and 
interacting with administrations and services, to cite only a few examples, more and more routinely involve 
the use of digital technologies. Over just two decades, microcomputers, laptops, mobile phones, and the 
Internet have provided users with powerful tools to search for and make use of immense repertories of 
information and services. Increasingly versatile mobile technologies allow users to stay connected almost 
regardless of where they are and what they are doing. And the integration of digital tools in homes, cars 
and appliances potentially increases the safety, flexibility, and effectiveness of many activities of everyday 
life. Navigating in unfamiliar environments, preventing domestic accidents, or encouraging the rational use 
of energy and water resources are just a few concrete benefits that can be expected from technological 
advances. 

3. Thus, early visionaries' dreams about a universal "information society" are, to some extent, 
coming true. But as computer technologies were pervading most areas of human activities, their actual 
impact on the development of human communities was found to be more ambiguous than initially expected 
(Forester, 1992). To start with, the spread of information technologies across the world is far from 
homogeneous. Despite two-digit growth rates reported in many countries at the dawn of the 21 st century, 
the distribution of digital technology uses across geographic and socioeconomic borders remains very 
uneven. Far from the rapid progress toward universal access envisioned by some a decade ago, as of May 
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2008, only one human in five had ever used the Internet (http://www.internetworldstats.com/). Personal 
access to computers and the Internet ranged from about zero in the poorest countries to 70% in the 
wealthiest ones. But even in those latter countries, access was still unevenly distributed across the 
socioeconomic spectrum, contributing to the so-called "digital divide" (Norris, 2001). 

4. Furthermore, access to computers and high-speed connectivity does not necessarily result in more 
social participation and well being. As surveys have revealed, heavy Internet use sometimes leads to a 
greater feeling of loneliness and less genuine interactions with others (Kraut et al., 1998). And, regardless 
of prior experience with technology, demographic factors such as age, level of income or education often 
correlate with users' ability to access and make effective use of computers in puiposeful contexts (Marquie 
& Baccarat, 1997; Sweets & Meats, 2004). 

5. Using computers or other digital devices to perform personal or work-related activities often 
presents a challenge for the everyday user. People often have trouble installing, setting up, and learning 
how to use new digital devices and software applications. Users often confine themselves to a few basic, 
but ineffective, procedures. Then, even routine computer use for mundane tasks is often prone to errors, 
delays and incidents. To take just one example, information searches using web sites often present complex 
problem solving situations for laypersons. Among other reasons, the nonlinear linking of information in 
hypertexts is often experienced as cumbersome or unpractical, yielding a sense of disorientation and 
cognitive overload (Rouet, 2006). Studies of information evaluation and integration also suggest that 
laypersons' source evaluation skills are challenged - and often overwhelmed. 

6. More experience on the part of users and improved design practices on the part of developers 
have been pointed out as important conditions for digital technologies to fulfill their potential. Though 
necessary, these are probably not sufficient conditions to make the online world accessible to all. There is 
a growing body of evidence that mere exposure to technology is not sufficient for people to achieve a 
satisfactory level of skill in purposeful technology-based tasks. So-called computer literacy skills must be 
integrated with deeper and more abstract problem solving skills (Lazonder & Rouet, 2008). This is because 
the most relevant uses of computers are not easily handled through the mere application of simple routines 
(e.g., launching an application and clicking on a set of buttons or links). Instead, to successfully complete 
computer-based tasks, people must be able to analyze the various requirements of the task, set up 
appropriate goals and plans, and monitor their progress through the task until the task purposes are 
achieved. This process seems best captured by the cognitive psychology construct of "problem solving", a 
construct that has been the subject of extensive research over the past forty years, in traditional as well as 
technological contexts. 

7. Analyzing the problem solving skills involved in the uses of digital technologies for various 
purposes and properly assessing the distribution of such skills in the general public appear to be two 
important conditions for the advancement of technology-rich societies. The data can be turned into high 
return education and training investments. These efforts, however, should be targeted towards relevant 
portions of the public, relevant educational levels, and relevant skills. The present framework aims to 
contribute to these important undertakings. 
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DEFINING PROBLEM SOLVING IN TECHNOLOGY-RICH ENVIRONMENTS 



8. In this section, we introduce a definition of problem solving in technology-rich environments. 
We first discuss the notion of problems and problem solving, focusing in particular on "information-rich" 
problems. Then we provide and explain the definition of PS-TRE to be used in PIAAC. 

Problems and problem solving 

9. A problem is usually defined as a situation where a person cannot immediately and routinely 
achieve his or her goals due to some kind of obstacle or difficulty. The ability to solve problems is 
considered one of the most complex and sophisticated aspects of human cognition (Newell & Simon, 
1972). In order to solve a problem, individuals have to first become aware of a difference between the 
current state of affairs and the state of affairs that would correspond to the satisfaction of their goals. In 
other words, they have to come to an understanding of the nature of the problem. This is also called 
"problem finding". Individuals then have to engage in a series of thinking processes and concrete actions in 
order to (a) define a set of sub-goals and steps through which the problem may be solved (also called 
planning or "problem shaping"), and (b) perform the actions required to reach those sub-goals until the 
situation reaches a satisfactory state. Throughout the problem solving activity, individuals have to monitor 
their progress and, where necessary, reconsider their goals and actions. For instance, individuals may face 
an unexpected outcome or find themselves at an impasse. In such cases, they may have to reconsider their 
understanding of the problem or the actions they have decided to take in order to solve the problem. 

10. Problem solving also normally requires a range of tools and information resources. In traditional 
areas, people may use source documents, paper and pencil, calculator and other devices. In the context of 
technology-rich environments, people may use Internet-based services such as search engines and Web 
pages, but also desktop software such as spreadsheets, email, or file management systems in order to solve 
their problem. Tools and technologies are normally meant to facilitate the resolution of the problem. They 
may, however, also contribute to making a problem more difficult, especially when a person has limited 
knowledge and experience with the use of those tools and technologies. 

11. In concrete, everyday situations, problems and problem solving often involve other persons. For 
instance, people may be asked to solve a problem for another person, they may need to get information or 
advice from another person, or they may want to communicate the solution to another person. The actions 
needed in order to solve the problem may then include spoken or written interpersonal communication 
(e.g., comprehending instructions, asking questions or explaining). Thus, communication skills must be 
considered a factor in assessing problem solving skills. In the context of technology-rich environments 
several powerful tools for rapid (e.g. mail and chat software) and broad (e.g. blogs, shared applications) 
ways of communication are available, thereby enabling collaborative problem solving activities by people 
being at different places. Such tools require special skills for computer-mediated communication 
(Bromme, Hesse & Spada, 2005). 

12. From a cognitive perspective, problem solving involves a complex hierarchy of processes and 
skills. The core characteristic of problem solving is that it is impossible for a person to achieve the goal 
through routine actions. In problem solving, one has to reflect on the situation in order to identify the 
proper arrangement of decisions and actions that may lead to a solution. Thus, the status of problems is 
conditional upon a person's familiarity with the problem or category of problems. With learning and 
practice, some activities that were initially experienced as problem solving may become routine activities. 
Examples include fundamental skills such as reading and performing mental calculations, as well as 
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everyday tasks such as tying one's shoes, replacing a broken light bulb, or installing new software on a 
personal computer. 

13. Regardless of a person's ability level, some problems are intrinsically more complex than others 
(Funke & Frensch, 2007). Dimensions of problem complexity include the clarity of the initial situation; the 
number of subgoals and steps needed to solve the problem; the amount of information to be considered; 
and the pragmatic constraints that surround the person's activity (e.g., time constraints, level of stakes or 
hazard, probability of unexpected events or outcomes). The complexity of a problem also varies as a 
function of the arrangement of informational and other resources in the problem-solving environment (i.e., 
extrinsic cognitive load, Sweller, Chandler, Tierney, & Cooper, 1990). For instance, reducing the distance 
between two pieces of information that need to be combined (e.g., a diagram and a legend) can make it 
easier for a person to solve a problem. 

14. Research on problem solving has also established distinctions between various types of problems. 
One important distinction is between closed vs. open problems. In closed problems the amount of 
resources (e.g., objects, tools) available and the range of possible actions is limited. An example is a chess 
game where moves are limited by the size of the chessboard and the rules of the game. In other problems, 
the potential resources and possible actions are, in principle, unlimited (Goel & Pirolli, 1992). For instance, 
finding one's way in an unfamiliar city or designing a new kitchen may be considered open problems. 

15. Another important distinction is between well-defined and ill-defined problems (Voss & Post, 
1988). Well-defined problems come with a set of circumstances that clearly let the person know what they 
are supposed to do. For instance, "Using the attached schedule, find a train to go from Paris to Amsterdam 
on Tuesday, October 15' h leaving no earlier than 11 a.m. and arriving no later than 9 p.m." would be 
considered a well-defined problem. In contrast, " Find a way to go to Amsterdam on Tuesday, October 15 lh " 
is an example of a less well-defined problem. It is important to note, however, that there is no 
straightforward link between the definition of a problem and its absolute level of difficulty. Sometimes ill- 
defined problems are easier to solve because they allow several solution paths. But ill-defined problems 
also require the problem solver to set up appropriate subgoals and operators, and to select appropriate 
resources, which may increase the difficulty of the problem. 

Information problems 

16. The distinction between information -rich and information-lean problems deserves particular 
attention in the context of PIAAC. Because digital technologies are primarily aimed at storing, processing, 
representing, and communicating symbolic information, the types of problems that will be used in the 
PIAAC problem solving assessment clearly belong in the first category. This contrasts, for instance, with 
logical or mathematical problems where the complexity lies in the computational reasoning, not so much in 
the information to be accessed and used. The PS-TRE domain of PIAAC is intended to cover the specific 
class of problems people deal with when using information and communication technologies (ICT). Those 
problems share the following characteristics. 

A. The existence of the problem is primarily a consequence of the availability of new technologies. 
One example relates to the vast amount of information now available on the World Wide Web. 
The Web has enlarged the access to specialist knowledge for laypersons. This gives rise to 
problems related to locating and evaluating information for quality and credibility, e.g., when 
seeking advice about legal issues or medical conditions (Stadtler & Bromine, 2007). Evaluation 
and critical judgment of information are core aspects of literate Internet use (Gilster, 1997) and 
will be one focus of the PS-TRE assessment. Other examples include the increasing capacity of 
electronic storage devices, with the subsequent problems of organizing and sorting large numbers 
of files; or the growing practice of social communication on the Web, with the subsequent 
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problem of learning and making use of new social norms as regards private vs. public 
information. 

B. The problem solution requires the use of computer-based artifacts (tools, representational 
formats, computational procedures) that were not available to the general public before the 
advent of personal computers. An example is the management of personal finance using 
spreadsheets, statistical packages and graphical tools. Here the problem itself may not be new 
(i.e., keeping spending in balance with income) but the new artifacts modify the distribution of 
work across social agents (professional vs. laypersons) and they deeply transform the procedures 
and steps required to solve the problem. 

C. The problems are related to the handling and maintenance of technology-rich environments 
themselves (e.g., how to operate a computer, how to fix a settings problem, how to use the 
Internet browser in a technical sense). 

17. Most of the problems that correspond to those broad characteristics require one to handle vast 
amounts of symbolic information. Therefore, they require an ability to deal with semantic content or 
meaning. Examples include understanding command names in drop down menus, the naming of files and 
folders, hits in a search engine, or links in a web page. Furthermore, many problems require the person to 
read and understand electronic texts, graphics and numerical data. Therefore, understanding and evaluating 
meaningful information available in technology-rich environments is central in the construct of PS-TRE. 

Definition 

18. In the context of the PIAAC survey, problem solving in technology-rich environments is defined 
as follows: 

"Problem solving in technology-rich environments involves using digital technology, 
communication tools and networks to acquire and evaluate information, communicate 
with others and perform practiced tasks. The first PIAAC problem solving survey will 
focus on the abilities to solve problems for personal, work and civic purposes by setting 
up appropriate goeds and plans, accessing and making use of information through 
computers and computer networks. " 

19. The two sentences in the definition each serve a specific purpose. The first sentence is aimed at 
providing a broad basis for the first as well as subsequent surveys of PS-TRE. The second sentence 
acknowledges some constraints that limit the scope of the first survey. We provide below a series of more 
specific comments on the words and phrases used in this definition. 

"using digital technology, communication tools and networks " 

20. PIAAC focuses on problems that are specifically related to the use of ICT. The problem solving 
context means that routine or basic ICT skills will not be central to the framework. Instead, PS-TRE will 
focus on situations that involve the active construction of goals and strategies on the part of the user. We 
also acknowledge the increasing diversity and versatility of digital technologies, and we emphasise that a 
proper assessment of PS-TRE should not be limited to traditional desktop computing. Instead, we envision 
that mobile and integrated technologies may be involved in new types of problem solving that will need to 
be represented in future assessments. 

"to acquire and evaluate information" 
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21. This phrase acknowledges that most uses of digital technologies involve the use of symbolic 
information, such as texts, graphics, links, and commands. Symbolic information is used as part of human- 
computer interfaces (e.g., icons, commands) and it constitutes the primary content of most computer 
applications (e.g., word processor, spreadsheet, Internet Browser, and email applications). The phrase also 
emphasises that computers and computers networks such as the Web mostly offer a multiplicity of 
information sources wherefrom (for the respective puiposes) the relevant and reliable pieces have to be 
chosen. 



''communicate with others" 

22. An important role of digital technologies is to provide powerful and flexible means for people to 
communicate with each other. Examples include email, chats, short message systems, and IP audio-visual 
communication. Digital communication may take place in the context of purposeful, problem-like 
situations and therefore it is an integral part of the PIAAC PS-TRE construct. 

"and perform practical tasks" 

23. The ability to solve problems with digital technologies is tightly related to the achievement of 
personal, civic and work-related purposes, which, in turn, take the form of concrete, practical tasks. 
Examples include shopping, learning about laws and regulations, and organizing teamwork through online 
agendas and reservation systems. The problems assessed in PIAAC will use authentic, meaningful 
scenarios based on surveys of computer uses and input from participating countries. 

"The first PIAAC problem solving survey". 

24. This is the first attempt to assess PS-TRE on a large scale and as a single dimension. This creates 
many challenges as regards the definition of tasks and the practical collection of data. Furthermore, digital 
technologies keep evolving at a rapid pace, as do the personal, social, and work-related uses of those 
technologies. While setting the stage for further rounds of surveys, the present framework will take a 
perspective on PS-TRE that takes into consideration feasibility issues as well as possible evolutions of 
technology and technology uses. 

"will focus on the abilities to solve problems for personal, work and civic purposes" 

25. In order to reflect the pervasiveness of ICT in the society, PIAAC PS-TRE will assess problem 
solving ability based on scenarios that pertain to these three important contexts. 

"by setting up appropriate goals and plans, " 

26. An assessment of problem solving capacity should focus on situations where test takers cannot 
immediately reach their goal based on routine, mechanistic sets of actions. Instead, we focus on tasks that 
require test takers to actively construct a solution based on the resources available in the assessment 
environment. 

"accessing and making use of information " 

27. Again, this phrase emphasises a specific aspect of PS-TRE, namely that these are often 
information-rich problems that require individuals to access, interpret and integrate multiple sources of 
information. 

"through computers and computer networks 



10 




EDU/WKP(2009)15 



28. There is more to "technology-rich environments" than merely personal computers. A full 
assessment of PS in TRE would require a range of devices that mimic the diversity and versatility of digital 
technologies of today's world. However, for feasibility reasons, this first survey will be limited to problems 
requiring the use of computers and Internet-based services. 



ORGANISATION 



Core dimensions of problem solving in technology-rich environments 

29. The domain of problem solving in technology-rich environments (PS-TRE) may be organised 
along three key dimensions (Figure 1). 

Figure 1 : Three core dimensions of problem solving in technology-rich environments 




Task / problem statement 
Elements of a situation that 
trigger and condition problem 
solving (eg, directions). 



Technolog ies 
The devices, applications 
and functionalities through 
which problem solving is 
conducted. 





Co gnitive dimensions 
The mental structures and 
processes by which a 
person actually performs 
problem solving. 



30. "Cognitive dimensions" involve the mental structures and processes by which a person actually 
performs problem solving. These include goal setting and monitoring progress; planning; locating, 
selecting and evaluating information; and organizing and transforming information. 

3 1 . "Technologies" are the devices, applications and functionalities through which problem solving is 
conducted. These include hardware devices (laptop computers for PIAAC), simulated software 
applications, commands and functions, and representations (text, graphics, and so forth). 

32. "Tasks" are the circumstances that trigger a person's awareness and understanding of the 
problem, and that determine the actions to be taken in order to solve the problem. In naturalistic situations, 
a wide range of conditions can initiate problem solving. For instance, a computer user may realise that their 
mailbox is crowded and that they need a new categorisation scheme; or they may need to reflect on a 
complex issue (such as finding out more about a medical treatment) and decide to look for information on 
the Web. In test-taking contexts, tasks are more explicitly assigned to participants. They include the 
question and task instructions presented to test takers, as well as the specific materials and time constraints 
associated with the test. 
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Cognitive dimensions 

33. Table 1 summarises the cognitive dimensions of problem solving that will be assessed in PIAAC. 
These are goal setting and progress monitoring, planning and self-organizing, acquiring and evaluating 
information, and making use of information. 



Table 1 : Cognitive dimensions in PS-TRE 



Dimension 


Examples 


Goal setting and progress 
monitoring. 


Identifying one's needs or purposes, given the explicit and implicit 
constraints of a situation 

Establishing and applying criteria for constraint satisfaction and 
achievement of a solution 

Monitoring progress 

Detecting and interpreting unexpected events, impasses and 
breakdowns 


Planning, self-organizing 


Setting up adequate plans, procedures, and strategies (operators) 
Selecting appropriate devices, tools or categories of information 


Acquiring and evaluating 
information 


Orienting and focusing one's attention 
Selecting information 

Assessing reliability, relevance, adequacy, comprehensibility 
Reasoning about sources and contents 


Making use of information 


Organizing information, integrating across potentially inconsistent 
texts and across formats, making informed decisions 

Transforming information through writing, from text to table, from 
table to graph, etc. 

Communicating with relevant parties 



Technology dimensions 

34. Table 2 summarises the technology dimensions that are taken into account in PIAAC. Hardware 
devices include artifacts that rely on digital technologies, such as desktop or laptop computers, mobile 
phones, and so forth. These devices are increasingly part of other devices, such as cars or homes, hence the 
phrase "integrated digital technologies". It is important to note that, regarding hardware devices, only 
laptop computers with simulated software applications will be included in the first cycle of this assessment. 
In addition, for operational reasons, sound, animations and videos will not be included. However, the 
general definition above provides for the inclusion of other digital devices in future cycles. 
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35. In addition to ai t i facts, technology-rich environments involve the use of software applications. In 

turn, these applications rely on commands, functions and representations of information. We list 
commands and functions independent from applications because some commands and functions may be 
found across a broad range of applications. Therefore, it is unclear if the knowledge of these commands is 
linked to that of particular applications where they may be found. Examples include "sort" or "find" 
commands. Similarly, texts, graphics and other representations are quite independent from the specific 
application where they may be found. 

Table 2: Technology dimensions of PS-TRE 



Dimension 


Examples 


Hardware devices 


Desktop or laptop computers, mobile phones, personal assistants, 
geographical information systems, integrated digital devices 


Software applications 


File management, Web browser, Email, Spreadsheet 


Commands, functions 


Buttons, Links, Textboxes, Copy/Cut-Paste, Sort, Find 


Representations 


Texts, Sound, Numbers, Graphics (fixed or animated), Video 



Note. Only laptop devices, a few simulated software applications and a restricted range of representations will be included in the first 
cycle of PIAAC. 

Task dimensions 

36. Table 3 summarises the dimensions of the tasks that will be assessed in PIAAC PS-TRE. These 
include the puipose and context in which the task is performed, the intrinsic complexity of the problem, 
and the explicitness of the problem statement and task directions given to the test taker. 

Table 3: Task dimensions in PS-TRE 



Dimension 


Examples 


Task purposes (contexts) 


Personal, Work/occupation, Civic purposes. 


Intrinsic complexity 


Minimal number of steps required to solve the problem 

Number of options or alternatives at various stages in the problem space 

Diversity of operators required, complexity of computation/transformation 

Likelihood of impasses or unexpected outcomes 

Number of constraints to be satisfied 

Amount of transformation required to communicate a solution 


Explicitness of problem statement 


Ill-defined (implicit, unspecified) vs. well-defined (explicit, described in detail) 



37. It is important to note that the "intrinsic complexity" of a problem is not a simple straightforward 
dimension. Intrinsic complexity may be characterised through a set of more specific dimensions, which 
include: the minimal number of steps or actions requested to solve the problem, the number of options at 
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each stage, the diversity of operators and the complexity of mental reasoning and/or computation, the 
probability of impasses or unexpected outcomes, the number of constraints to be satisfied, and the amount 
of composition or transformation required in order to communicate a solution. 

Number of steps or sub-goals required to reach a solu tion 

38. Tasks which present a problem with a single goal and few required steps are likely easier than 
those with multiple goals or sub-goals which require a number of steps to reach a solution. 

Probability of impasses or unexpected outcomes 

39. Tasks that include unanticipated impasses or outcomes are expected to be more difficult. One 
advantage of a computer-delivered assessment of problem solving is the possibility of building tasks in 
which additional constraints or outcomes can be introduced as a test taker works through a task. For 
example, at a defined point in a task, an unexpected email might display, adding new information that test 
takers must take into consideration while working towards a problem solution. 

Amount of transformation and generation required to communicate a solution 

40. Tasks requiring test takers to represent or compose information to convey a solution would likely 
be more difficult than tasks with more defined responses. Examples of transformation and generation tasks 
include constructing a table, re-representing text in a graph, or writing a justification. In order to keep the 
PS-TRE framework distinct from numeracy, we do not include the production of statistical graphs in the 
tasks. Neither do we include the writing of lengthy open justifications, because of difficult scoring issues. 
We may, however, ask participants to evaluate the communicative effectiveness of a graph or to select 
among several possible justifications. 

Specificity and explicitness of task constraints 

41. It is expected that tasks which explicitly define the problem to be solved and the steps required to 
reach a solution will be easier than tasks which present ill-defined problems. A problem situation that 
requires the selection of operators, sub-goals, or to define successful achievement of a goal makes the 
problem more difficult. 

Problem solving in technology-rich environments vs. other domains of PIAAC 

42. The constructs of literacy, numeracy, and PS-TRE rely on the same “core” cognitive processes. 
For instance, the ability to decode printed symbols and at least a minimal working memory capacity are 
required for tasks in any of these domains. PS-TRE, however, will assess a set of competencies that are 
distinct from the other two constructs. Aspects that distinguish PS-TRE from the other domains include the 
following. 

• Problem solving specifically assesses goal setting, monitoring, and planning in technology-rich 
environments. Therefore PS-TRE tasks will emphasise the processes of problem finding and 
problem shaping that are typical in these environments. Problem solving tasks will include 
selecting an appropriate software application, deciding on a strategy among several possible, 
making use of adequate functionalities in a context-sensitive manner, interpreting ill-structured 
texts and making use of online forms. 

• Problem solving tasks will be carried out in environments that involve multiple, complex sources 
of information. Some of the tasks will require the test taker to use multiple environments and to 
shift across environments. PS-TRE will therefore assess decision-making with regard to 
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information sources to be used (for example, the act of choosing which environment to use or 
whether or not to go to another web site.) Evaluation will be included as a critical underlying part 
of problem solving. Additionally, selecting appropriate devices or tools will take a prominent role 
for this domain. 

43. In terms of information processing, problem solving is a specific construct in that: 

• It focuses on the pragmatic evaluation of information (and most notably the assessment of 
sources) in terms of reliability and adequacy relative to the problem at stake, as opposed to the 
mere topical relevance of text passages, which is more applicable to the literacy construct. 

• It focuses on the integration of information across multiple sources. Integration may require the 
person to deal with differences in contents, genres, langage levels across sources; to update their 
knowledge when a new source provides more accurate information; or to acknowledge that 
different sources diverge when the sources provide discrepant information about a situation. 

44. PS-TRE tasks will be kept as simple as low as possible on numeracy and literacy demands in 
order to increase the specificity and validity of the construct. 

Problem solving in technology-rich environments vs. other related constructs 

45. Problem solving in technology-rich environments is related to several other constructs that have 
been designed either as part of international surveys or in other contexts. We point out the similarities and 
differences between PS-TRE and some other constructs below. 

46. The expert group addressed the issue of how the problem-solving domain is differentiated from 
the general ICT domain. ICT skills may be broadly defined as "the interest, attitude, and ability of 
individuals to appropriately use digital technology and communication tools (...)" (Lennon et al., 2003). It 
is acknowledged that, just like literacy and numeracy, ICT skills underlie PS-TRE. However, the PS-TRE 
construct aims to go beyond purely instrumental skills related to the knowledge and use of digital 
technologies. The cognitive dimensions of problem solving are considered the central object of the 
assessment, with the use of ICT as secondary. 

47. Distinctions between PS-TRE and the Big6™ Skills also exist. The Big6™ skills model is 
defined as an information and technology literacy model and curriculum (Eisenberg, 2008). It rests on the 
acknowledgement that information problems involve a series of core steps: namely task definition, source 
selection, location and access of content information, extraction of content information, synthesis, and 
evaluation. Although logically sequential, the steps do not necessarily follow each other linearly during 
complex information problem solving. The Big6™ model is meant to describe a wide range of information 
problems, among which are problems requiring the use of ICT. The Big6™ model, however, does not 
specifically focus on tasks that are novel or complex (i.e., actual problems), as the PS-TRE framework 
does. 
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OPERATIONAL ASPECTS 



48. Part 4 reviews some practical aspects of the assessment, especially regarding the evaluation of 
problem solving strategies in the context of digital technology. 

Prerequisite skills 

49. Achievement of PS-TRE tasks presupposes the mastery of foundational 1CT skills. These 
include skills associated with manipulating input and output devices (e.g., the mouse, keyboard, and digital 
displays), awareness of concepts including files and folders, and an understanding of basic file 
management operations such as save, open, close, delete, move, and rename. In addition, test takers should 
be at least minimally familiar with simple graphical interface features, such as the iconic representation of 
files and folders, hyperlinks, scrollbars, and different types of menus and buttons. 

Task development 

50. Test developers will use the defined task characteristics, which include cognitive dimensions, 
technology environments, and contexts, to build and code tasks for the problem solving assessment. 
Assuming the development of 25 tasks for consideration for the field test, it is recommended that the 
proportion of tasks be as shown in Table 4. Additionally, the distribution across contexts is recommended 
to be 40% personal, 30% occupational and 30% civic. 



Table 4: Distribution of tasks as a function of environment and cognitive dimensions 





Web 


Spreadsheet 


Email 


Multiple 




9 


4 


6 


6 


Goal setting and monitoring 
progress 


2 


1 


1 


1 


Planning 


2 


2 


2 


4 


Acquiring and evaluating 
information 


3 


0 


0 


0 


Making use of information 


2 


1 


3 


1 



51. In addition, the task dimensions of intrinsic complexity and explicitness of the problem 
definition, as listed in Table 3, will be included as variables during development as they are expected to 
influence the difficulty of items in the problem solving assessment. 

Item development goals 

52. The test design for PIAAC specifies the number of scenarios needed for the problem solving 
assessment, as shown in Table 5. 
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Table 5: Problem solving development targets 





Development Pool 


Field Test 


Main Assessment 


Total 


25 task proposals 


16 scenarios 
(5, 10, or 15 minutes) 


8 scenarios 
(5, 10 or 15 minutes) 



53. It is anticipated that the three types of scenarios listed will vary in terms of specific 
characteristics including: the number of required cognitive processes (e.g., goal setting and monitoring; 
planning; acquiring and evaluating information; making use of information); the number and kind of steps 
required to complete a task; the inclusion of unexpected outcomes or impasses to which a test taker must 
respond; and the extent to which tasks are open-ended or explicitly broken down into a series of defined 
steps. The indication of 5, 10 and 15 minutes arise from global constrains on the design and application of 
the PIAAC survey and do not reflect any theoretical rationale as to the correspondence between cognitive 
processes and time on task. Furthermore, we anticipate large variations in completion time, partly in 
relation to prior experience with computers and overall level of skill. Below are some more specifications 
about each type of scenario: 

• "Five-minute scenarios" may range from easy to more difficult but they are envisioned as the 
most directed and least complex of the tasks in the assessment. For example, a test taker may be 
given a simulated page of hits from an Internet search and be asked to evaluate the choices and 
select one that meets a short set of specified criteria. 

• "Ten-minute scenarios" will likely involve multiple steps and, in some cases, multiple technology 
environments. For example, test takers might be presented with a problem which requires them to 
locate an email message, open an attachment, and then use information in the attachment to 
create a brief table that will represent the information for a specific purpose. 

• "Fifteen-minute scenarios" will be designed to simulate real-life problem-solving activities that 
are recursive and more exploratory in nature. The tasks will often cross multiple environments, 
and require test takers to employ several, if not all, of the cognitive components. A sample 
complex scenario would be one in which a test taker has to conduct a search in the simulated web 
environment, integrate and evaluate information across a number of sites and then use the 
information to generate a summary to be shared as part of a community presentation. 

Capturing and scoring performance indicators 

54. Skilled problem solving includes the ability to effectively reach problem-solving goals, and to do 
so using the most efficient combination of means and actions. Therefore, the assessment of PS-TRE skills 
requires both a measure of problem-solving performance and a measure of strategy effectiveness. 

55. The cognitive components that underlie PS-TRE (Table 1) all contribute to skilled performance. 
The competent problem solver is presumably good at setting goals, planning, acquiring and making use of 
information. The components may nevertheless be related to different underlying cognitive abilities. For 
instance, goal setting may depend on a person's reasoning ability, whereas locating information may 
depend on the person's visual scanning and reading ability. A fine-grain assessment of problem solving 
skills should be based on a broad set of indicators of the underlying cognitive components. The constraints 
of a large-scale international assessment, however, do not allow the application of the relevant fine-grain 
measurement procedures. Therefore, only overall indicators of problem solving performance and strategy 
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will be collected in PIAAC. The manipulation of task and environment characteristics across items will 
ensure that all the required cognitive components will be assessed. 

56. A critical aspect of the test development process will be defining the actions to be captured by the 
software and determining construct-based scoring criteria. The test delivery software will be able to collect 
more than the output or product of a participant’s response to a task. For any given task, the computer can 
collect a variety of information including time spent, actions taken, and the sequence in which actions are 
completed. This information provides direct evidence of the processes and strategies participants use to 
solve the presented problems, and therefore it improves the inferences we can make about their knowledge 
and skills. Given the possibility to capture process and strategy information, it is anticipated that the survey 
will yield an extensive amount of raw data. It is therefore important to use the framework variables, the 
hypotheses about task characteristics that will impact performance, and information from the cognitive labs 
to define those behaviors that provide the best evidence about performance and are therefore important to 
capture and score. 

57. It is anticipated that the scoring models developed around each scenario and task will allow for a 
range of data to be aggregated into variables that can be used to understand performance across the 
problem solving scale. This will include those aspects of performance that are associated with fulfilling the 
task demands as well as those related more to strategies and processes that are undertaken. Subject matter 
experts including task designers will provide initial input into the creation of these scoring models. These 
models will be reviewed and changes made as appropriate based on feedback from reviewers, the cognitive 
labs and field test data. The goal is to use the raw data that is captured to create a set of variables that can 
reliably capture and distinguish performance along the problem solving scale. 
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